The present invention is related to log stream processing for applications, sensors or instruments.
Most modern day automated software's generate human readable logs for developers to understand and realize the cause of errors, or progress of transactions in the application. This is standard practice usually by developers of the application itself to assist in debugging possible scenarios that may arise in real-life. While logs are present in most-modern day systems, the scale of connected components and devices makes finding the root-cause an extremely hard search problem. The current practice of manually mining the root-cause is arduous, relies on the operators understanding of the system, and can take several man-hours.
Conventional systems for finding production bugs such as Splunk, Loggly, and ELK require manual inspection. Splunk is a search engine for traditional log inputs. These systems index all words in logs collected from production systems, and provide a Google like search capability. This is simple yet extremely powerful, as often operators can simply find root cause by searching for key-words and manually correlating causes across various sources. Loggly and ELK go a step further and provides tokenization of logs based on regular expressions given by users, or taken from a repository of commonly available regular expressions for common applications. The tokenized logs are then indexed and stored in a database, which supports search. Both of these provide real-time alerts and visualizations to the users.